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Purpose and Overview 



Purpose : To provide education officials, program providers, and others seeking to sponsor 
a rigorous program evaluation with practical advice on finding a capable evaluator. 

More specifically, this Guide offers practical advice on finding an evaluator to conduct a rigorous 
evaluation to measure the effect of an educational program on outcomes such as student 
achievement. (Such evaluations are often called "impact" evaluations.) The Guide's main focus is 
on finding an evaluator to conduct a well- designed randomized controlled trial 1 , since that is the 
primary type of evaluation capable of meeting the evidence standards of the What Works 
Clearinghouse of the U.S. Education Department's Institute of Education Sciences (IES). However, 
the Guide also includes endnotes in each section discussing how the principles might be adapted 
for those seeking to sponsor other high-quality evaluations when a randomized controlled trial is 
not feasible (e.g., well- matched comparison- group studies or regression-discontinuity studies). 

This Guide limits itself to key principles, and does not seek to be an exhaustive list of factors to 
consider in finding a capable evaluator. 

Problem the Guide seeks to address : Many attempts at rigorous evaluation fail to yield credible 
estimates of a program’s effect because of serious flaws in study design or implementation. 

In some cases these flaws mean that a program evaluation, despite a considerable investment of 
effort and funds, does not produce a valid answer to the basic question of whether the program 
has a meaningful effect on educational outcomes. 

Often, these are flaws that a highly-capable evaluator could have foreseen and prevented. 

Illustrative examples of common flaws include: (i) using a sample too small to detect meaningful 
effects of the intervention; (ii) failing to obtain and analyze outcome data for a high proportion of 
the original sample; and (iii) measuring surrogate outcomes that lack practical and policy 
importance (e.g., attitudes toward school, rather than attendance and graduation rates). 2 

To address this problem : We suggest finding an evaluator with two core capabilities: 

(i) A demonstrated track record in conducting well-designed randomized controlled trials (or 
another type of high-quality study design if that is what you plan to sponsor); and 

(ii) Expertise in the program area being evaluated, or closely-related areas. 

The reason we recommend this approach is that an evaluator's demonstrated track record in 
conducting high-quality evaluations is likely to be a stronger predictor of success in conducting 
your study than, for example, anything the evaluator might promise to do on a piece of paper, as 
part of an application. We also recommend finding an evaluator with substantive expertise in the 
program area being evaluated (e.g., K-2 math education), or closely-related areas, so as to ensure 
that the study selects sites, sample members, outcome measures, and other design features that 
are appropriate for the program being evaluated. 

The remainder of this Guide offers step-by step advice on how to find an evaluation team 
with these core capabilities. The Guide assumes you already know the general parameters of the 
study you are sponsoring (e.g., main questions you seek to answer, study timeline, available budget). 

It then outlines a process for finding a capable evaluator that is streamlined so as minimize cost and 
administrative burden. If you have questions as you go forward, please contact the What Works 
Clearinghouse Help Deskat 1-866- WWC-9799, info(a)whatworkshelpdesk. ed. gov . 



3 




Step 1 - Identify a pool of candidates with experience conducting randomized 
controlled trials in program areas similar to yours. 3 



Here are several possible approaches you might take, separately or in combination, to 
identify this initial pool of candidates: 

■ Search the What Works Clearinghouse’s Registry of Outcome Evaluators . 

The Registry can be an efficient tool for building your initial pool of candidates because it 
enables you to (i) search for and screen evaluators by content area (e.g., dropout prevention), 
geographic availability, and organizational size; (ii) view evaluations in which they have been 
involved and whether these evaluations used an experimental design (i.e., a randomized 
controlled trial); and (iii) in many cases, download reports on these evaluations. Keep in mind, 
however, that the evaluators on this list are self-nominated, so there is no guarantee that the 
information they provide is accurate, nor that the randomized controlled trials they list are of 
high quality (hence the importance of the vetting process in step 2, below). 

* Find the authors of previous well-designed randomized controlled trials in program areas 
similar to yours. 

To find authors of well-designed trials, you can go to websites that summarize findings from 
such trials, such as the What Works Clearinghouse or other sites listed in Appendix A of IES's 
Identifying and Implementing Educational Practices Supported By Rigorous Evidence. We 
suggest you look specifically for randomized controlled trials that these sites identify as 
meeting the highest standards of evidence, and find the list of study authors, who in most 
cases played a central role in the study. Some of these sites also list the authors' contact 
information, to facilitate your following up with them. 

■ Ask around for recommendations of evaluators with a demonstrated track record in 
conducting well-designed randomized controlled trials in program areas similar to yours. 

For example, you might ask for such recommendations from (i) individuals identified through 
the methods listed above who are not themselves available to conduct your study; (ii) research 
organizations with expertise in randomized controlled trials, such as the IES's Regional 
Educational Laboratories and the organizations listed in Appendix A of the IES Guide; (iii) 
professional associations of researchers that specialize in such studies, whose websites and 
meeting proceedings can be found on-line; and (iv) schools, districts, state agencies, local 
universities, and other entities who you may know have sponsored, conducted, or participated 
in such studies. 

As you get started, you can obtain personalized assistance in navigating the above resources from 
the What Works Clearinghouse Help Desk ( 1-866- WWC- 9799, info(a)whatworkshelpdesk. ed. gov) . 

■ Issue a solicitation for capability statements from potential evaluators. 

Government and other organizations that fund studies and/ or program services often solicit 
capability statements as a first step in sponsoring a project. The solicitation, in this case, 
could be a very simple statement describing the evaluation project you will be sponsoring, and 
asking potential evaluators to submit evidence of a demonstrated track record in conducting 
well-designed randomized controlled trials in program areas similar to yours. The evidence 



4 




they submit should include (i) study reports on two randomized controlled trials they have 
conducted, at least one of which is in a program area similar to yours; and (ii) the resumes of 
key staff who would work on your study and their expected roles. 



Step 2 - Review the randomized controlled trials conducted by your initial 
candidates, to determine whether they were well-designed and implemented. 



For the candidates identified in step 1 who are interested in conducting your study, we suggest the 
following actions: 

Ask them to send you the items in the suggested capability statement above - namely: 

(i) The study reports on two randomized controlled trials 4 they have conducted, at least one 
of which is in a program area similar to yours; and 

(ii) The curricula vitae of key staff who would work on your study, and their expected roles. 

We then suggest, as a threshold matter, that you briefly peruse their submission to confirm (i) 
whether the studies they submitted were, in fact, randomized controlled trials; and (ii) whether 
one or more of their proposed key staff helped conduct these trials. Once you have confirmed 
these items, we suggest the following action. 

Have each candidate’s two trials reviewed briefly by 1-2 experts to determine whether 
they were well-designed and implemented. 5 

■ Why we believe this step is critical : As noted earlier, many randomized controlled trials 
fail to produce credible results because of serious flaws in design and/or implementation. 

This is true even of many trials published in peer-reviewed journals. It is therefore important 
for you to verify independently whether your candidate evaluators have the demonstrated 
ability to avoid such flaws and conduct a study that produces valid evidence. 6 

■ The expert(s) you ask to review the trials submitted by the evaluator(s) should have a key 
qualification: the ability to read a randomized controlled trial and judge its quality. 

We suggest, therefore, that you look specifically for individuals with a strong understanding of 
randomized controlled trial research, although they need not necessarily be a researcher 
themselves or intimately familiar with your program area. If you don't already have such an 
expert in your organization, you might engage an outside expert who has this key qualification 
to conduct the review, perhaps as a brief consulting project. Examples of organizations that 
may have, or be able to recommend, experts in randomized controlled trial research include 
IES's Regional Educational Laboratories and the organizations listed in Appendix A of the IES 
Guide. For personalized assistance in navigating these resources as you get started, contact 
the What Works Clearinghouse Help Desk ( 1-866- WWC- 9799, info(a)whatworkshelpdesk. ed. gov) . 

■ We suggest you ask the expert(s) to conduct a brief (e.g., 30-60 minute) review of each 
submitted study, using the 10 “key items” listed on pages 5-9 of the IES Guide . 

The IES Guide lists 10 key items to look for when reviewing a randomized controlled trial to see 
whether it was well-designed and implemented. You should make clear to the expert(s) that 
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